System and method for transforming dispersed data patterns into moving objects

ABSTRACT

A motion-based method and system for rapidly identifying the presence of spatially dispersed or interwoven patterns in data and their deviation from a test model for the pattern includes transforming dispersed patterns into one concentrated moving objects, for which there is a characteristic, identifiable motion signature. The method may be used with data sets containing sharp peaks, such as frequency spectra, and other data sets. A roadmap of basic motion signatures is provided for reference, including multiple harmonic series, separation of odd and even harmonics, missing modes, sidebands and inharmonic patterns. The system and method may also be used with data stored in arrays and volumes. It remaps such data to show both high-resolution information and long range trends simultaneously for applications in nanoscale imaging.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of priority of U.S.Provisional Patent Application Ser. No. 60/900,715 entitled “A GraphicalSystem and Method for Identifying and Characterizing Spectra ContainingHarmonics” filed on Feb. 12, 2007. The present application also claimsthe benefit of priority of U.S. Provisional Patent Application Ser. No.60/907,552 entitled “Graphical Technique for Exploring SpectraContaining Harmonics and Other Spectral Patterns” filed on Apr. 6, 2007.The present application also claims the benefit of priority of U.S.Provisional Patent Application Ser. No. 60/979,124 entitled “AGraphical, Motion-Based Method for Exploring Spectra ContainingHarmonics, Sidebands, and Other Patterns” filed on Oct. 11, 2007. Theseprovisional applications are incorporated by reference in theirentirety.

FIELD OF THE INVENTION

The present invention is directed to a motion-based method and systemfor rapidly identifying the presence of patterns and deviations in datarepresentations. Particularly, the present invention is directed toenhancing visual recognition of systematic effects upon data patternsusing motion enhancement techniques to identify simultaneouslyhigh-resolution characteristics and long-range trends.

BACKGROUND OF THE INVENTION

A rustle in the bushes or a movement in the grass that cannot be audiblyperceived provides visual cues based upon the motion of the leaves ormovement of the grass. Image representations, such as shapes, size,color, position, and others, effectively prime the pre-attentive processof the human visual system.

Visual items often consist of regions that differ in color, luminance,size, and shape. The human visual system is adept at binding togetherthese various regions to perceive the whole object, while simultaneouslyseparating the regions from those that belong to other objects or to thebackground. The human visual system parses the regions and recognizeswhole objects. Motion cues play a significant role in the process ofrecognizing objects. Objects that are substantially similar to otherobjects within a field of vision are more difficult to discern. Motioncues enable integration of regions within an object as well as theability to separate the object from the background. Additionally, motioncues facilitate development of object representations, which permitrecognition in static images. Motion information plays a fundamentalrole in organizing the visual experience and in assisting a viewer inboth integrating regions within an object and in separating one objectfrom another or from a background.

Standard data displays do not fully utilize the exceptional ability ofthe human visual processing system to respond to motion cues. Moreover,standard techniques often display patterns with 1) spatialinterruptions; or 2) spatially separation in the display. Long,stretched out representations of data are not ideal for the visualidentification or separation of multiple, interwoven patterns. Even inthe case of a single pattern, small systematic deviations from a trialmodel for the pattern become hard to judge from an extended,stretched-out representation. Paradoxically, a standard x-y plot canexhibit these problems even when the underlying data contains elegantsymmetry. These representation problems may be indicative that thevisualization technique itself may be unnecessarily encumbering theprocesses of data exploration, hypothesis generation and analysis.Addressing this type of visualization bottleneck ameliorates a viewer'sability to explore and present global trends in data, and to avoidlimiting the presentation to a small, localized window when comparingexperimental data with modeled data behaviors.

Dynamic graphics may be incorporated into data analysis, includingdynamic regression analysis. However, for data involving stronglyinterwoven patterns, the residuals with respect to a model for thepattern become difficult to calculate. To calculate these residuals,numerical algorithms have been used. However, by introducing numericalalgorithms as an intermediate step in the analysis, there are risks forerror. For example, when using numerical algorithms, one might make apriori assumptions about trends in the data, or introduce errors inperforming peak fitting. A direct, rapid way to produce the dynamicregression plot without calculating the residuals is needed.

The extent to which an imperfect visual display is relied upon in theprocess of in setting up the numerical algorithm creates a circularproblem, and is also frequently overlooked. An advanced repertoire ofnumerical algorithms exists for analyzing data containing patterns ofpeaks. For example, such algorithms may be applied to audio and speechanalysis. These algorithms often operate by reducing the data to asingle number or perhaps a few numbers (such as a set of fundamentalfrequencies), but in doing so hide complex relationships in the data. Inthe case where knowledge of the trends in the data that contribute tothe final output is important to the user, reduction of the data to asingle number overprojects the data. As a result, when working with thenumerical algorithms, such as in debugging, in developing, and in otherenvironments, one typically uses an x-y plot of the raw data in anattempt to monitor the trends, which in turn produces the bottlenecksdescribed above, namely spatial separation and spatial interruption ofpatterns. Some numerical algorithms produce results that are directlyshown in x-y plot form, such as cepstrum, autocorrelation, and the like,and these processes exhibit these same problems. Data analyzed via theshort time Fourier transform are presented in 2-D. However, patterns arespatially separated along the y-axis.

Mappings may be used to transform visual proximity. In acoustics andpsychoacoustics research, mappings of pitch onto cyclic curves are usedto increase the proximity of members of a given pitch class. A varietyof different mappings have been developed, from the Pythagorean spiralof fifths to the helical representation of pitch defined by Drobisch aswell as those discussed by Ruckmick, and in the study and family ofmappings constructed by R. N. Shepard, which includes a threedimensional toroidal map. Pikler presented a historical perspective ofpitch-related computations utilizing spirals and reviewed the work ofPtolemy and others. Pikler also reported new developments andexperimented with the imaginary domain. More recently, Chew created the“Spiral Array”, and has explored pitch spelling applications. However,all of these techniques focus on musical pitch and tend to spatiallydisperse patterns in spectral harmonics, as they are optimized for adifferent set of applications. Further, these techniques do not includemotion to organize the visual experience and to assist a viewer in bothintegrating regions within an object and in separating one object fromanother or from a background.

Another manner of increasing the proximity of stretched-out serial datais raster scanning. Raster scanning is an example of transforming serialdata so that proximity of stretched-out patterns is increased. Earlywork performed by Lashinsky produced raster plots of spectra using anexperimental setup that integrated a spectrum analyzer with anoscilloscope, using the latter to produce the final display. Lashinsky'stechnique seeks periodicity in spectral data and was used to displaypatterns of harmonics in spectra. However, this scheme not only does notincorporate motion, but removes it through its triggering method. Forsufficiently tangled spectra, static displays can leave patterns hiddeneven when they remap the data to increase proximity.

When a frequency spectrum or block of data is presented in serial form,real effects may exist within the spectrum or block of data that aredifficult for the eye to perceive. Graphical transformations of the datamay better illustrate these effects and enhance the process of dataexploration and algorithm development. Long, stretched-out serial visualpatterns are easier to recognize and compare as compact objects.

Motion cues facilitate development of object representations, butefforts to date have not successfully incorporated motion with compactobjects to provide recognition.

Efforts to date to improve the ability to recognize and characterizepatterns within data presentations have been largely unsuccessful inproviding graphical techniques that provide visual recognition ofsystematic frequency effects and data patterns. Efforts aimed atimproving the ability to accurately identify harmonics, spectralfeatures, and other data patterns have not provided satisfactoryresults. What is needed is a system and a method for accuratelyidentifying data patterns using compact data and motion enhancement toprovide visual recognition of frequency effects and data patterns.

SUMMARY OF THE INVENTION

The system and method of the present invention spatially compactspatterns in data sets and transforms them to moving objects in a visualdisplay, thereby changing the manner in which they are recognized. Whenthe objects move, they become easier to recognize. The present inventionincorporates a graphical transformation of data sets, including serialdata and blocks of data, including arrays of data, and volumetric data.The system and method of the present invention is used to addressvisualization bottlenecks that arise when patterns are spatiallyinterwoven and/or spatially separated in a display.

Frequency spectra are an example of a data set that often containsinterwoven spectral patterns. The spectra of polyphonic music, forexample, have overtone sequences that interrupt each other whenpresented sequentially. The appearance of sidebands around families ofharmonics produces similar interweaving of related groups of spectralpeaks. Even for a single overtone series, the relative change inseparation between a pair of partials close to the fundamental and apair of partials at a higher frequency is difficult to judge because thelines or segments to compare sit far apart in the display. As such,spectra serve to illustrate visualization bottlenecks as spectralpatterns are spatially interrupted or spatially separated with respectto the viewer.

Outputs of numerical algorithms that are typically presented in the formof a contour plot (such as the Short Time Fourier Transform, forexample) and microscopy images (such as those taken with electron orscanning probe microscopes, as examples) are examples of data sets thatare saved as arrays of data and may contain patterns that are interwovenand/or spatially dispersed in the display. Nanoscale microscopy imagesillustrate a particular kind of visualization bottleneck. The choicemust often be made between a high resolution image, in which long rangetrend information is lost, or a low resolution image which contains thetrend information, but perhaps displays it in a visually inefficientmanner.

The system and method of the present invention is used to analyze suchdata and to demonstrate a manner in which compacting the data set andemploying motion effects is used to untangle patterns in the data.Specifically, the system and method of the present invention is used toanalyze spectra and other data sets, including serial data, arrays ofdata, and volumetric data, and demonstrates compacting the data set andemploying motion effects to untangle spectral and other data patterns.

The system and method of the present invention extends data recognitiontechniques and provides an improved view of the distributions ofpatterns after spatially compacting the data and transforming patternsinto moving objects for improved recognition.

The system and method of the present invention offers an efficientmanner of rapidly producing a dynamic regression plot directly fromserial data without calculating the residuals that avoids a number ofsources of error introduced by other techniques. When calculation ofresiduals is bypassed, the number of a priori assumptions needed tocreate the plots is reduced. The index (i.e. partial number) associatedwith each peak in the data does not need to be known in advance. Thisoffers significant advantages when several types of patterns, each withits own set of indices, are interwoven. The technique differs fromresidual plotting in that multiple reference lines are used—not just thex-axis. Additionally, non-uniform mappings may be used to produceadditional reference curves. The system and method of the presentinvention introduces redundancy that is not normally present in aresidual plot, but the redundancy can be optionally avoided by limitingthe plot range. Further dimensions of information are included than in atypical residual plot. Namely, the information of intensity as well asthe peak positions are also contained in the plots

Signatures of effects associated with the spectra and with the harmonicsare identified and characterized. Multiple harmonic series in thespectra, odd and even harmonics, and missing modes are mapped ininterpretable forms in a rectangular coordinate system. For example, thespectra and the harmonics may be mapped in an x-y plane, in athree-dimensional rectangular coordinate system, in a polar coordinatesystem, and the like. For applications in audio signal processing, themethods of the present invention may be used in tandem with quantitativetechniques, such as comb filtering and other fundamental frequency andpolyphonic pitch identification methods.

A method of the present invention is used to analyze data sets and tountangle patterns in the data by compacting the data set and employingmotion effects to visualize the data patterns. The system and method ofthe present invention is used to analyze spectra and other data sets,including serial data, arrays of data, and volumetric data. For example,a method of the present invention transforms a frequency spectrum dataset from a frequency space to an animated coordinate space, such as arectangular coordinate system, a polar coordinate system, and the like.The method includes mapping a frequency axis onto a curve, mapping avalue of a function of the frequency spectrum data set onto a thirddimensional axis of the rectangular coordinate space, and scanning themap to produce an animated contour plot sequence in the rectangularcoordinate space.

Mapping the frequency axis onto a curve may include replicating thefrequency spectrum into a plurality of lines, aligning the plurality oflines in parallel in the rectangular coordinate space such that the lastline intercepts the horizontal axis of the rectangular coordinate spaceat a horizontal coordinate equal to the number of the plurality oflines, and shifting each line by an amount that is a function of thenumber of lines in the plurality of lines to map a target spectralsequence along the horizontal axis of the rectangular coordinate space.

The curve may be displayed in an x-y plane with a horizontal x-axis anda vertical y-axis, as well as in other coordinate systems. Additionally,the vertical axis may be limited to avoid redundancy in the contour plotsequence. The target spectral peak sequences may be mapped along thex-axis with additional target spectral peak sequences positionedconsecutively along the x-axis. Further, the value of a function offrequency may be a Fourier transform amplitude, phase, power, spectraldensity, or other values. The lines may be of equal length and may beshifted by an amount that is a function of the frequency function.Additionally, the lines may be aligned vertically.

A example system in accordance with the present invention transforms afrequency spectrum data set from a frequency space to an animatedcoordinate space, such as a rectangular coordinate system, a polarcoordinate system, and the like. The system includes a frequency axismapping module that maps a frequency axis onto a curve, a frequencyfunction mapping module that maps a value of a function of the frequencyspectrum data set onto a third dimensional axis of the coordinate space,and a scanning module that scans a parameter to produce an animatedcontour plot sequence in the coordinate space.

The frequency axis mapping module may include a curve definition modulethat replicates the frequency spectrum into a plurality of lines, wherethe plurality of lines including at least a first line and a last line,and a shifting module that aligns the plurality of lines in parallel inthe rectangular coordinate space such that the last line intercepts thehorizontal axis of the rectangular coordinate space at a horizontalcoordinate equal to the number of the plurality of lines and whereineach line is shifted by an amount that is a function of the number oflines.

The frequency axis mapping module may display the curve in an x-y planewith a horizontal x-axis and a vertical y-axis, as well as in othercoordinate systems. Additionally, the frequency axis mapping module mayfurther map target spectral peak sequences along the x-axis and positionadditional target spectral peak sequences consecutively along thex-axis.

Also, the curve definition module may replicate the plurality of linesinto equal lengths to normalize the frequency axis. Additionally, thefrequency function mapping module may map the value of a function offrequency in the frequency spectrum that is Fourier transform amplitude,phase, power, spectral density, or other values.

In accordance with one embodiment of the present invention, the systemand method of the present invention compacts data. Data that is afunction of a single variable is mapped onto a curve in the x-y plane sothe target sequence in the data appears along the x-axis. Parametricequations are used to describe the transformation. The method oftransforming the frequency spectrum from a frequency space to anothercoordinate system may be implemented by mapping this data multiple timesonto a set of parallel lines and shifting each line so that itintercepts the x-axis in a target sequence of shift values. In the casewhere the data is a function of frequency (such as Fourier transformamplitude, phase, power spectral density, etc.) may be assigned to thez-axis. Scanning the definition of the target sequence of shift valuesproduces an animated contour plot sequence that is readily perceptibleand may be efficiently identified and characterized.

When the data is in the form of an array of values, the method oftransforming the array may be implemented by mapping the array multipletimes so that it is centered about the set of parallel lines, andshifting each array so that it intercepts the x-axis using a targetsequence of shift values. When the data is in the form of a volume ofvalues, each volume is mapped multiple times so that it is centeredabout a set of parallel lines in two dimensions, and each volume isshifted so that it that intercepts the x-y plane using a target array ofshift values. Scanning the definition of the target array of shiftvalues produces motion.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate an embodiment of the invention anddepict the above-mentioned and other features of this invention and themanner of attaining them. In the drawings:

FIG. 1 is a schematic illustration of a data transforming system inaccordance with one embodiment of the present invention.

FIG. 2 illustrates scan sequences to simulate movement of referenceobjects formed by harmonics of a fundamental frequency in accordancewith an embodiment of the present invention.

FIG. 3 illustrates scan sequences to simulate the motion of inharmonicspectral sequences in an embodiment of the present invention.

FIG. 4 illustrates scan sequences to simulate the motion of sidebands inspectral sequences in an embodiment of the present invention.

FIGS. 5A-5C illustrate the identification of two harmonic series in aspectrum in accordance with an embodiment of the present invention.

FIG. 6 illustrates a challenge spectrum power spectral density as afunction of frequency as an x-y plot in accordance with the presentinvention.

FIGS. 7A-7C illustrate a residual plot, a transformed power spectrum,and a power spectrum, respectively, of a B^(b) octave in accordance withthe present invention.

FIG. 8A depicts a comparison of odd and even harmonics in accordancewith the present invention.

FIG. 8B graphically illustrates the characteristics of the odd and evenharmonics in accordance with the present invention.

FIGS. 9A-9F illustrate the identification of inharmonic tails bytracking even harmonics across frames of a scan in accordance with thepresent invention.

FIG. 10 shows plots of an inharmonic target spectral sequence usingscanning methods of the present invention to produce motion.

FIG. 11 shows a comparison of an octave before and after distortion andthe corresponding spectra using a method of the present invention.

FIG. 12 shows data with sidebands generated using an odd nonlinearfunction in accordance with the present invention.

FIG. 13A illustrates a spectrum and analysis using visual polyphonicpitch identification in accordance with the present invention.

FIG. 13B illustrates a spectrum produced by a concert grand piano and acorresponding analysis in accordance with the present invention andcompares it to a spectrum produced by a digital keyboard and itscorresponding analysis.

FIG. 14 illustrates scan sequences to view groups of sinusoids andcorresponding spiral features in accordance with the present invention.

FIG. 15 illustrates an underlying sinusoid and an alias sinusoid andcorresponding spiral mappings in accordance with the present invention.

FIG. 16 illustrates harmonic tracking in a spectral analysis of an audiosignal in accordance with the present invention.

FIG. 17 shows high resolution images of an optical grating data setacquired with an scanning electron microscope and a transformed image inaccordance with the present invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a schematic illustration of a data transforming system 10 inaccordance with one embodiment of the present invention. In accordancewith the illustrated embodiment, the data transforming system 10 isprovided with a frequency axis mapping module 20 that is adapted to mapa frequency axis onto a curve. Frequency axis mapping module 20 includesoptional curve definition module 30 and shifting module 40. Curvedefinition module 30 is adapted to replicate a frequency spectrum into aplurality of lines, including at least a first line and a last line.Shifting module 40 is adapted to align the last line vertically in arectangular coordinate system such that the last line intercepts thehorizontal axis of the rectangular coordinate system at a horizontalcoordinate equal to the number of lines. Shifting module 40 is alsoadapted to center the lines about the horizontal axis of the rectangularcoordinate system.

Additionally, data transforming system 10 includes frequency functionmapping module 60 adapted to map a value of a function of the frequencyspectrum onto a third dimensional axis of the rectangular coordinatesystem. Data transforming system 10 also includes scanning module 50adapted to scan the frequency spectrum to produce an animated contourplot sequence in the rectangular coordinate system.

Data transforming system 10 of FIG. 1 may be implemented with any typeof hardware and software, and may be a pre-programmed general purposecomputing device. For example, the data transforming system 10 may beimplemented using a server, a personal computer, a portable computer, athin terminal, a hand held device, a wireless device, or any combinationof such devices. The data transforming system 10 may be a single deviceat a single location or multiple devices at a single location, ormultiple devices at multiple locations that are connected together usingany appropriate communication protocols over any communication mediumsuch as electric cable, fiber optic cable, any other cable, or in awireless manner using radio frequency, infrared, or other technologies.

In the illustrated embodiment, the data transforming system 10 afrequency transforming system and is connected to a network 999 thatallows remote access to the frequency transforming system so thatfrequency information may be entered on terminals 14 a, 14 b and so thatanalysis may be conducted by users 16 a, 16 b that are remote from thefrequency transforming system. This configuration also provides anefficient manner in which users 16 a, 16 b may collaborate by sharingdata and results of analyses. Additionally, frequency spectra data andanalysis results may be stored in database 70 to provide searchable andretrievable records with which users 16 a, 16 b may perform additionalanalyses or may be used to verify additional measurements.

The network 999 allows the data transforming system 10 to interact withterminals 14 a, 14 b, which may be proximate to data transforming system10 or which may be remotely located from data transforming system 10.Regardless of their location, terminals 14 a, 14 b are utilized by users16 a, 16 b. The network 999 may be any type of communications channel,such as the Internet, a local area network (LAN), a wide area network(WAN), direct computer connections, and the like, and may be connectedin a wireless manner using radio frequency, infrared, or othercommunication technologies, using any type of communication hardware andprotocols. The terminals 14 a, 14 b may be implemented using a server, apersonal computer, a portable computer, a thin terminal, a hand helddevice, a wireless device, and any other computing devices, or anycombination of such devices.

It should also be noted that the data transforming system 10 inaccordance with one embodiment of the present invention is illustratedand discussed herein as having a plurality of modules and/or componentsthat perform particular functions. It should be understood that thesemodules are merely schematically illustrated based on their function forclarity purposes only, and do not necessary represent specific hardwareor software. In this regard, these modules and/or sub-modules may beimplemented as hardware and/or software to substantially perform theparticular functions explained. Moreover, two or more of these modulesmay be combined together within the data transforming system 10, ordivided into more modules based on the particular function desired.Thus, the present invention as schematically embodied in FIG. 1 shouldnot be construed to limit the data transforming system 10 of the presentinvention.

In the above regard, the data transforming system 10 in accordance withthe illustrated embodiment also includes an interface module 80 thatallows the users 16 a, 16 b to easily enter frequency spectra data andperform analyses on the frequency data via terminals 14 a, 14 b.Similarly, interface module 80 provides an input and output pathway fromusers 16 a, 16 b and data transforming system 10. In this regard, theinterface module 80 includes a graphical user interface (GUI) forreceiving frequency data and other information input by users 16 a, 16b, and a GUI for displaying the analysis information for the variousfrequency data input.

Databases are provided in the data transforming system 10 in accordancewith the illustrated embodiment. In particular, the data transformingsystem 10 shown includes a generic database 70, which includes storedfrequency spectra data as well as analysis data, analysis tools,filters, and other data and files that may be used to perform actions onthe frequency spectra data. Other information regarding the frequencyspectra can be stored in the database 70 as well.

FIGS. 2-16 show example data sets using frequency spectra and comparethe resulting compact forms of the spectra with the originalstretched-out forms as the spectra are transformed using the system andmethod of the present invention.

FIGS. 17A and 17B show additional analysis of an array of data valuesusing the system and method of the present invention. The array analyzedin FIG. 17B was formed by concatenating multiple high resolution imagessuch as that in FIG. 17A to form a new, panoramic image. Thetransformation of dispersed data patterns in the panoramic image intomoving objects employs the method and system of the present invention tosimultaneously view high resolution information and macroscopic trends.The two-dimensional transformation example utilizes microscopy imagesfrom a scanning electron microscope.

Mapping Definition

In accordance with one embodiment of the present invention, the systemand method of the present invention compacts a spectrum from its serialform to increase the proximity of spectral patterns to form visualobjects, and the objects are made to move to enhance recognition.

The procedure used to produce the visual display may be implemented bymapping a frequency axis multiple times onto a set of parallel verticallines in the x-y plane, shifted in such as way that a target spectralpeak sequence lies along the x-axis. The set of parametric equationsshown below (Equations 1) sets the peak sequence F_(n) along the x-axis,where n is the integer index and F_(test) is a test fundamentalfrequency. The value of a function of frequency, F, (such as Fouriertransform amplitude, phase, power spectral density, etc.) is assigned tothe z-axis.

$\begin{matrix}{{Equations}\mspace{14mu} 1} & \; \\\begin{matrix}{x = n} \\{y = {\frac{1}{F_{test}}\left\lbrack {F - F_{n}} \right\rbrack}}\end{matrix} & (1)\end{matrix}$

Motion may be produced by varying F_(test) or another parameter. Thechoice of non-uniform mappings in x and y, such as log-log andlog-linear, further extends the method of the present invention.

A useful special case occurs when the target spectral sequence is theharmonic series, Fn=nF_(test). Then, Equations 1 reduce to:

$\begin{matrix}{{Equations}\mspace{14mu} 2} & \; \\{{x = n}{y = {\frac{F}{F_{test}} - n}}} & (2)\end{matrix}$where n is the harmonic number and consecutive harmonics of the testfundamental, F_(test), lie along the x-axis, separated by one unit each.Restriction of the harmonics to the horizontal strip, −½≦y<½, avoidsredundancy. The method of transforming the spectrum effectivelyreplicates the spectrum into lines. Normalizing F to F_(test) gives eachline or segment a length of one.

FIG. 3 illustrates the motion of an inharmonic spectral sequence, andhow changing the target spectral sequence and scan parameter definitionscan be used to produce different motion signatures. Each column of scans310, 320 compare the positions where {circumflex over (F)}={circumflexover (n)}F₀√{square root over (1+{circumflex over (n)}²B)} for twodifferent target spectral sequence models. In scan 310, test model 311is the series that would appear along the x-axis and is F=nF_(test) andF_(test) is scanned through F₀. In scan 320, test model 322 is given byEquation 3 below, where F_(test)=nF₀ is fixed, and B_(test) is scanned.Scan 330 is a log-log version of scan 320.

FIG. 3 illustrates the motion of an inharmonic spectral sequence, andhow changing the target spectral sequence and scan parameter definitionscan be used to produce different motion signatures. Each column of scans310, 320 compare the positions where {circumflex over (F)}={circumflexover (n)}F₀√{square root over (1+{circumflex over (n)}²B)} for twodifferent target spectral sequence models. In scan 310, test model 311is the series that would appear along the x-axis and is F=nF_(test) andF_(test) is scanned through F₀. In scan 320, test model 322 is given byEquation 5 below, where F_(test)=nF₀ is fixed, and B_(test) is scanned.Scan 330 is a log-log version of scan 320.

As briefly described above, FIG. 4 illustrates the motion of sidebands.The positions of a few sidebands 410, 420, 430 around harmonics of F₀₁are shown. Scans show snapshots as F_(test) is scanned through F₀₁,using Equations 2. In this case, each family of sidebands 410, 420, 430pivots about its center, that is one of the harmonics of F₀₁. ScanningF_(test) through 2F₀₁ separates the families of sidebands 410, 420, 430around the odd and even harmonics of F₀₁. Another signature is producedwhen F_(test) is set to the separation between sidebands. When F_(test)is set to the separation between sidebands, F₀₁ and F₀₂, each family ofsidebands 410, 420, 430, pivots about its center as shown in FIG. 4.Other useful values of F_(test) for sum and difference frequenciesgenerated by Fo1 and Fo2 include F_(test)=(Fo1+Fo2)/2, or eitherfundamental. Then, the motion signatures resemble those of FIG. 2,except that a grid of sidebands pivots along with the line of harmonics.

Relationship with Residual Plotting

Equations 1 take a frequency spectrum, another function of frequency, ormore generally, any serial data set and map it so that the peaks in thedata automatically appear in the positions of a residual plot withrespect to the target spectral sequence that is defined for the x-axisintercepts. Residual plots provide excellent results when comparing themagnitude of deviations between a measurement set and model predictionbecause the eye's bias when judging the magnitude of deviations isdifferent when the reference is a straight line than when it is a curve.In addition to the advantage of bypassing the process of calculatingresiduals, Equations 1 go beyond standard residual plotting techniquesin several ways. A third dimension of information (the peak height) isincluded through the color map. Furthermore, Equations 1 can producemore than one meaningful reference curve, as can be seen in FIGS. 2, 3,and 4. The additional reference curves in FIGS. 2, 3, and 4 often appearalong linear features in the x-y plane in useful cases, whichfacilitates the visual interpretation of systematic deviations.

Generation of Companion Maps with Complementary Features

In accordance with the present invention, a number of alternatives toEquation 1 and 2 can be defined, including concentric circles orspirals. More companion maps can be produced by taking (x,y) in Equation1 or 2 to be in the complex plane and applying conformal mappingtechniques. The circular or spiral maps have the advantage that patternsof harmonics join at the origin, making their presence easier to detectbecause of the increased visual continuity. However, Equations 1 and 2offer the advantage over other maps of similarities with the standardtechnique of residual plotting in special cases, as described above.Since the multiple reference curves in FIGS. 2, 3, and 4 often appearalong linear features in the x-y plane, their corresponding referencecurves in a new set of roadmaps obtained via conformal transformationsare found by following how a line transforms in each map.

A harmonic series is a rudimentary example that illustrates the basicprinciple of the mappings. Data containing two harmonic series isanalyzed as an illustration before analyzing more complicated data sets.

Visual identification of two different harmonic series in data occursthrough the spatial regrouping of the positions of harmonics ofdifferent fundamentals. When a spectrum containing energy at harmonicsof an unknown fundamental, F₀, is analyzed by scanning F_(test) inEquation 2, consecutive harmonics lie along lines with a slope thatdepends on the difference of F₀ and F_(test). When F_(test) is scannedthrough F₀, the row of harmonic positions appears to pivot about theorigin, sweeping through the x-axis when F_(test)=F₀. A priori knowledgeof the exact value of a fundamental frequency in the data is notrequired.

An example of the separation and identification of two harmonic serieswith different fundamentals is shown in FIG. 5A. The spectrum 530 of aminor second (pitches A and B^(b) for which the fundamentals F₀ are 65.2Hz and 69.0 Hz, respectively) played simultaneously with a 16 foot organpedal flute stop is shown in FIG. 5C. The two second duration signal wasrecorded with a sampling frequency, F_(s), of 44.1 kHz and 16 bitresolution. The transformed data for F_(test)=65.2 Hz is shown in FIG.5B, where the harmonics of 65.2 Hz group together along the x-axis andthose related to 69.0 Hz lie along the oblique line 540, which wrapsaround the selected strip of the x-y plane near the 8^(th) harmonic.

Identification of Basic Effects Using a Challenge Spectrum

The challenge spectrum chosen as the main test data set is that of anoctave played on a piano. This particular spectrum was chosen foranalysis because it challenges many pitch identification algorithms,which often suffer from octave ambiguity problems. The upper note of theoctave adds energy to the even harmonics for low partial numbers. Knowneffects in piano spectra include inharmonicity, phantom partials, andcoupled string effects.

The spectrum 630 shown in FIG. 6 was obtained from a B^(b) octave, thatis, a lower B^(b) of F₀₁=58.0 Hz and an upper B^(b) of F₀₂=2×58.0 Hz.The B^(b) octave was played fortissimo on a grand piano with full pedal.The spectrum 630 was recorded at 96 kSa with 24 bit resolution. ABlackman window weighting was applied to the 1.5 second long signal toreduce leakage within the Fourier Transform analysis.

Following a brief discussion of the advantages and disadvantages of themapping in comparison with a family of related maps, the transformedversions of the spectrum 630 shown in FIG. 6 are juxtaposed with thetraditional x-y plot form in the following sections.

Analysis of the inharmonicity of piano tones illustrates a number offeatures of the mapping technique in accordance with the presentinvention. Equation 3 below models overtones, F_(n), associated with apiano string.F _(n) =nF ₀√{square root over (1+n ² B)}  (3)

Equation 3

In Equation 3, B is the inharmonicity coefficient. The inharmonicitycoefficient depends on string-related parameters, and thus varies acrossa piano. When the harmonic series is used as the model, inharmonicityappears in a residual plot 710 as a systematic deviation from thehorizontal reference axis for large integers n as shown in FIG. 7A. Whenthe FFT amplitude or power spectrum 730 is transformed for the caseF_(test)=F₀ deviations from the harmonic series positions automaticallyappear as in the residual plot 710. Note that no peak fitting has beenperformed and no assumptions about the correspondence of peaks with aparticular value of n were required to create the plot.

The transformed data 720 shown in FIG. 7B for F_(test)=58.0 Hz revealthe signature of several distinct inharmonic spectral tails 722, 724.Because the inharmonicity parameter, B, varies across the keyboard, thevalue of B for the lower B^(b) is different than the value for the upperone. Since the lower B^(b) contributes to both the odd and evenovertones while the upper B^(b) adds energy to the even overtones, theability of the system and method of the present invention to show thedistribution of the tails between odd and even spectral peaks is veryvaluable.

When the test fundamental is twice the fundamental of a harmonic seriesin the data, the odd harmonics 825 and even harmonics 815 separate inspace, grouping along the horizontal lines at y=0 and y=−½,respectively, as shown in FIG. 8A. This enables further examination ofthe imbalance in the distribution of odd and even harmonics and isuseful in resolving octave ambiguity.

Continuing the analysis of the spectrum shown in FIG. 7C, thetransformed data of the power spectrum 730 corresponding to F_(test)=2F₀is shown in FIG. 8B, confirming that two of the inharmonic tails 835,845 appear only in the odd harmonics.

Separation of Every m^(th) Harmonic

When the test fundamental frequency, F_(test), is exactly an integermultiple, m, of F_(o), every m^(th) harmonic lies on one of m equallyspaced horizontal lines in the horizontal strip, with each consecutiveinteger x-position filled with a harmonic. Rational F_(test)/F_(o),produce a similar signature, but have gaps between harmonic positions inx. When viewed in animated form, these signatures appear in acharacteristic temporal sequence, creating a visual rhythm as objectsappear and disappear in the display, which greatly helps to visualizeand identify the presence of a particular harmonic series.

Motion for Feature Identification

Once the frequency spectrum is compacted as described above, the systemand method of the present invention employs motion to improvevisualization of the frequency spectrum and the frequency effects.Overlap due to wrapping effects in the right half of contour plots makethe tails of the even harmonics difficult to differentiate. ScanningF_(test) introduces motion into the analysis. The frame sequence 910,920, 930, 940, 950, 960 in FIGS. 9A-9F run from F_(test)=116.3 Hz to119.1 Hz and have been extracted from a longer F_(test) animation.Motion facilitates the identification of features of the spectrum.

The frame sequence 910, 920, 930, 940, 950, 960 suggests that theremight be four tails 932, 934, 936, 938 rather than three tailsassociated with the even harmonics 431 (best shown in FIG. 9C). Trackingthe even harmonics across the frame sequence 910, 920, 930, 940, 950,960 of an F_(test) scan assists in the visualization of four inharmonictails. Odd harmonics 941 have wrapped around and re-emerged from the topof the horizontal strip (best shown in FIG. 9D). When viewed in animatedform, the tails 932, 934, 936, 938 change curvature at different rates,and this relative motion in the right half of each image of the framesequence 910, 920, 930, 940, 950, 960 helps to visualize this effect.The animated frame sequence untangles four rather than three tailsassociated with the even harmonics for the spectrum of the octave. Theeffects may be quantified using pitch identification techniques. Ofcourse, other quantitative measures may also be employed. When appliedin tandem with comb filtering or other pitch identification techniques,the technique and method of the present invention may be used tointerpret the output of the pitch identification techniques. Thesetechniques are described in further detail in the sections below.

Inharmonic Model for Target Spectral Pattern

To reset the target spectral sequence to an inharmonic model, the modelgiven by Equation 3 is used for F_(n) in Equation 1 with theinharmonicity parameter, B, replaced with a variable test parameter,B_(test). Motion is introduced by scanning B_(test) and is shown in FIG.10 for F_(test)=2F₀. Comparison of the behavior at y=0 and y=½ confirmsthat two of the tails are only associated with the even harmonics. Atlarge n, six prominent branches, with indications of sub-groupings,become visible in the final frames of FIG. 10. For small n, the lastframes 1010, 1020 of FIG. 10 show agreement with the target model,Equation 3, but the cluster 1022 shows a general, systematic deviationfrom the model along the x-axis for large n. The motion-based featuresof the technique provide a tool for displaying the detail of the globaltrends over larger ranges of n than before, reaching the portions of thespectrum where the x-y plot becomes hard to interpret directly. Thiseffect may be visualized by comparing FIG. 10 with FIGS. 6 and 9.

Provided that F_(test) is set to the fundamental frequency of the pitch,when the model accurately represents the data, B_(test) is scannedthrough the actual value of B and the partials related to thisparticular tail appear along the x-axis. In this special case,deviations can be interpreted as residuals. However, when F_(test) isnot equal to the fundamental frequency of the pitch, deviations areverified for each choice of target spectral sequence model, in additionto partial residual plots.

For example, when F_(test)=2F₀ as in FIG. 10, because this is thespectrum of an octave, F_(test) is set to the fundamental frequency ofthe upper B^(b) but is off from the fundamental frequency of the lowerB^(b) by a factor of 2. Equation 3 with n replaced by n/2 shows that inorder for a tail associated with the lower B^(b) to appear along thex-axis, B_(test)=4B. In FIG. 10, deviations may also be present if thepiano is out of tune.

Synergistic Use with Quantitative Techniques

Synergistic pairings can be made between the visualization technique andquantitative numerical techniques, such as the cepstrum,autocorrelation, Short Time Fourier Transform, comb filtering techniquesand the like. When a comb filter is used for pitch identification,output is produced whenever the comb filter passbands coincide withspectral peaks. Generally, the global maximum of the comb filter outputis the fundamental frequency. Comb filters produce subsidiary, localmaxima when the passbands coincide with every m^(th) harmonic. Pairedwith the visualization system and method of the present invention, boththe subset of spectral peaks that act to produce the subsidiary, localmaxima and the subset that does not contribute can be tracked at thesame time, providing a different point of view of the comb filteroutput.

When used in tandem with existing inharmonic comb filtering techniques,the scan shown in FIG. 10 may be used to determine the origin of localmaxima in the inharmonic comb filter output when the inharmonicityparameter is scanned, and may be useful in optimizing the inharmonicmodel chosen for a given piano.

Sidebands

Two examples involving sidebands are shown in FIGS. 11 and 12. When theaudio signal 1111 corresponding to FIG. 6 was passed through anover-driven loudspeaker, the distortion-related peaks 1121 appeared asshown in FIG. 11. The data in FIG. 12 was synthetically generated usinga nonlinear function, y(x), and an input, x(t) consisting of two tones:x(t)=A cos(2πF ₁ t)+A cos(2πF ₂ t)y(x)=tan h(x)  (4)

Equations 4

with A=2, F₁=395 Hz, and F₂=405 Hz and is plotted so that the evenharmonics of 400 Hz would lie along the x-axis, and the odd harmonicsand associated sidebands group around y=−½ and y=½.

Example Visual Pitch Identification in Polyphonic Music

When the spectrum of an excerpt of polyphonic music is displayed, thepartials (overtones) related to each pitch appear in non-consecutiveorder in the spectrum, making relationships between partials difficultto judge, especially for high partial numbers. The spectrum of theovertone cluster produced in the aftersound at the end of the firstphrase of the Copland Piano Variations (1932) is analyzed here as theexample. The signal, acquired from a Steinway concert grand, was of onesecond duration and was sampled at 44.1 kHz. Note the difficulty ofvisually extracting the patterns directly from the spectrum in serialform, shown in FIG. 13. In FIG. 13, snapshots 1313, 1323 extracted froma dynamic scan exhibit organized energy along the x-axis whenF_(test)=65.4 Hz and F_(test)=138.6 Hz, indicating energy at theharmonics of both fundamentals. In snapshot 1323, the odd and evenpartials related to the lower pitch also appear in diagonal 1326. Oneapplication of the system and method of the present invention is in thedevelopment of synthetic musical instruments, such as digital keyboards,in which it may be useful to compare side-by-side visualizationsanalyzing the sound produced by a synthetic instrument withcorresponding visualization for the sound produced by the realinstrument. Another application is in the improvement or replication ofmusical instruments or in comparing the quality of different musicalinstruments of the same type. In FIG. 13B, an example of a spectrumproduced by a concert grand piano 1370 is analyzed for two values ofF_(test) and shows the presence of overtones with fundamentals near 65.4Hz and 138.6 Hz. The same spectrum 1380 and analysis 1385 is shown for adigital keyboard. The analyses allow details of the discrepancies insound to be compared.

As shown in FIGS. 2-13, complex spectra can be explored effectively withthe system and method of the present invention to compare details of thediscrepancies in sound.

The system and method of the present invention identifies harmonics andother frequency spectra effects by spatially compacting the spectrum anduses motion of the resulting waveforms to facilitate identification andcharacterization of related sets of peaks in the plots. The aboveexamples from audio spectra of musical instruments illustrate thesignatures of multiple harmonic series, systematic deviation from theharmonic series, and differences between odd and even harmonics.Additionally, complex spectra may be transformed from a visuallyindigestible serial form to a form that reveals information about theharmonic content. Different functions of frequency were utilized in theexamples, including Fourier transform amplitude, power spectrum, and thelike. Other options include comparison of the phases of differentharmonics or periodic modulations in the envelope of spectral peaks.Other applications include treating frequency, F, in Equation 1 as adummy variable to extend the methods of the present invention to datadependent on space, time, and other variables. The system and method ofthe present invention may be used in examining and analyzing spectradata and in generating hypotheses and conclusions regarding the spectraboth alone, and in tandem with quantitative techniques.

Tracking Harmonics and Artifacts in Spectra Using Sinusoidal and SpiralMaps

For example, in accordance with one embodiment of the present invention,a method is directed to developing expressions for the positions of theobjects in the case where the moving objects are formed from periodicpeaks in the data. Mapping the data onto a sinusoidal curve centers theperiodic peaks in the data at the same positions as a sampled sinusoid.By defining transformations that map the sinusoidal curve onto othercurves, the behavior of a sampled sinusoid can then be used to predictthe positions and movement of the periodic peaks in the new map.Examples of a few simple curves include lines, concentric circles,spirals, helices, and the like. An example of a mapping of the sinusoidonto lines consists of the lines defined by Equation 1 in the horizontalstrip between y=−½ to y=½, where each line segment corresponds to oneperiod of the sinusoid. Alternatively, Equation 2 may also be used aswell. A similar example of a mapping onto concentric circles maps eachperiod of the sinusoid onto one of the concentric circles. A similarexample of a mapping onto a spiral sets one turn of the spiral to oneperiod of the sinusoid. This example of a mapping onto a spiral sets isillustrated in detail below to demonstrate the system and method of thepresent invention by which the combination of maps can be used to deriveproperties of the new map. The combination of maps can be used todevelop an interpretation of new curves that cut across the originalcurves. The latter plays a role in both perceptual (characteristic ofthe human visual process) and non-perceptual (numerical) aspects of themethods.

As an example, a spatial separation of a specific musical pitch from anartifact is demonstrated below to highlight features of the system andmethod of the present invention.

Synopsis of Approach

To demonstrate the efficacy of the system and method of the presentinvention, the below example utilizes a pair of spectral maps. Themethod of the present invention starts with the systematic set ofpatterns that appear when a sinusoid is sampled near the Nyquist rate. Amapping that concentrates these patterns is chosen. The sinusoid istransformed to a spiral. A linear, rather than a logarithmic map of thespectrum is used, and the spiral is defined in the complex plane. Aprocedure for mapping a spectrum onto a sinusoidal curve is defined. Thepositions of harmonics in this map match with those of samples of thesinusoid. The relationship between the two maps and a sampled sinusoidfacilitates the derivation of properties. The utility of curves that cutacross the underlying sinusoid and spiral for the application ofharmonic tracking is then shown. Further, the linear map of frequencyoffers advantages for the application of harmonic tracking and isdemonstrated with a practical example using an audio spectrum.

Mapping Definitions and Properties

Revisiting the Patterns Formed by Sinusoid Samples Near the Nyquist Rate

When a sinusoid is sampled close to the Nyquist rate, patterns appearthat compete for attention with the underlying sinusoid. This effect isan example of a phenomenon often referred to as “visual dissonance.”Sampling just above the Nyquist rate produces these perceptual effectsin plots, despite a sampling rate sufficiently high to unambiguouslydetermine the frequency of the underlying sinusoid. When viewing asinusoid sampled near the Nyquist rate, the eye tends to track patternsaccording to spatial proximity. The first column 1401 of FIG. 14displays the systematic evolution of these patterns 1410 a, 1410 b, 1420a, 1420 b, 1420 c as the sampling is scanned from below the Nyquist rateto above the Nyquist rate. Groups of sinusoid samples 1415, 1425, 1435,1445, 1455 match directly with spiral features 1416, 1426, 1436, 1446,1456. When the frequency axis is mapped along the underlying curves,spectral harmonics center about the former positions of the samples, sothat the patterns 1410 a, 1410 b, 1420 a, 1420 b, 1420 c shown in FIG.14 can be linked with either groups of samples or groups of relatedharmonics. The unsampled sinusoid is given by Equation 5 below:

$\begin{matrix}{{{s(u)} = {\cos\frac{2\pi}{a}u}};} & (5)\end{matrix}$and the sampled sinusoid is given by Equation 6 below:

$\begin{matrix}{{{s(n)} = {\cos\; 2\pi\frac{T_{s}}{a}n}},} & (6)\end{matrix}$where Ts=1/Fs is the sampling period.Concentrating the Patterns

Principles of Gestalt psychology indicate that the perception of thesevisual effects can be altered by increasing the spatial proximity ofpoints belonging to one of the patterns while dispersing the pointsbelonging to another pattern. This can be accomplished using the systemand method of the present invention by transforming the sinusoid into adifferent curve characterized by periodicity in space. Theconcentrating/dispersing effect obtained by transforming the sinusoidonto an Archimedean spiral is shown in the second column 1402 of FIG.14, where the mapping is given by Equation 7:

$\begin{matrix}{\theta = {\frac{2\pi}{a}u}} & (7) \\{{{sinusoid}:{s(\theta)}} = {\cos\;\theta}} & (8) \\{{{spiral}:z} = {{r\;\angle\;\theta} = {{r\;{\mathbb{e}}^{j\theta}} = {a\;{\theta\mathbb{e}}^{j\theta}}}}} & (9)\end{matrix}$

The spiral has been defined in the complex plane with polar coordinates(r, θ). The period, a, of the sinusoid determines the constant ofproportionality of the spiral. Both underlying curves given by Equations7 and 9 become visually distinguishable when the sinusoid issufficiently oversampled. The regime for examination for this instanceis close to the Nyquist rate for the sinusoid.

Spectral Maps

Spectral Map Definitions

To map a spectrum along either of these curves, the curves areparameterized in frequency using Equation 10.θ=2πF/F _(test)  (10)

As such, the parameterized sinusoid and spiral become Equation 11 andEquation 12, respectively:

$\begin{matrix}{{{sinusoid}:{s(F)}} = {\cos\; 2\pi\frac{F}{F_{test}}}} & (11) \\{{{spiral}:{{r(F)}{{\angle\theta}(F)}}} = {2\pi\; a\frac{F}{F_{test}}{\mathbb{e}}^{j\; 2\pi\frac{F}{F_{test}}}}} & (12)\end{matrix}$

Spectral intensity and other functions of frequency occupy a thirddimension, such as a color map coordinate or height above the curve.

Positions of Harmonics

When a pure harmonic series with fundamental frequency, F_(o), residesin the spectrum, the positions of the harmonics, F_(n)=nF_(o), along thesinusoidal map ideally center about the same positions as the sinusoidsamples. The positions of the harmonics, F_(n), in both maps can befound from Equation 13 and Equation 14.

$\begin{matrix}{{{sinusoid}:{s(n)}} = {\cos\; 2\pi\frac{F_{o}}{F_{test}}n}} & (13) \\{{{spiral}:{{r(n)}{{\angle\theta}(n)}}} = {2\pi\; a\frac{F_{o}}{F_{test}}n\;{\mathbb{e}}^{j\; 2\pi\frac{F_{o}}{F_{test}}n}}} & (14)\end{matrix}$

Equations 6 and 13 are in the form of Equation 15, such that:s(n)=cos 2πfn  (15)where f is a normalized frequency. Comparing the two equations(Equations 6 and 13) yields the following relationship between thesampling parameters and the harmonic tracking parameters,

$\begin{matrix}{f = {\frac{T_{s}}{a} = \frac{F_{o}}{F_{test}}}} & (16)\end{matrix}$Special Cases

Special case 1. When F_(test)=F₀, the harmonics of F₀ appear at thepositive crests of the sinusoid and at the intersections of the spiralwith the positive real axis in the complex plane.

Special case 2. When F_(test)=mF_(o), where m is a positive integer, theharmonics of F_(o) appear along straight radial arms in the complexplane which run along the same angles as those of the m^(th) roots ofe^(j2πn). In the sinusoidal map, this angle corresponds to a phaseshift. Every m^(th) harmonic appears along the intersection of one of mhorizontal lines with the underlying sinusoid.

Aliasing

Sinusoid Sampled Below Nyquist Rate

The Nyquist rate is just satisfied when the sampling period, T_(s)=a/2,or, equivalently, F_(test)=2F₀, so that there are two samples persinusoid cycle. Continuing special case 1 from above, if F_(test) isbelow F_(o), the sinusoid falls into the range where aliasing willoccur. More precisely, if F_(test)=F_(o)(1−α), where |α

, the alias frequency can be calculated by returning the normalizedfrequency, f=F_(o)/F_(test), to the range −½<f<½:

$\begin{matrix}{{s(n)} = {\cos\; 2{\pi\left( {\frac{F_{o}}{F_{test}} - 1} \right)}n}} & (17) \\{\mspace{45mu}{= {\cos\; 2{\pi\left( {\frac{1}{1 - \alpha} - 1} \right)}n}}} & (18) \\{\mspace{45mu}{\cong {\cos\; 2{\pi\alpha}\; n}}} & (19)\end{matrix}$

The alias, s(n)=cos 2πu/â can then be found from Equations 16 and 19,and the alias period is approximately â=a/α.

The period, a, of the underlying sinusoid fixes the constant ofproportionality, a, of the underlying spiral curve. The alias period, â,also defines a spiral curve 1516 in the complex plane at r

θ=aθe^(jθ) as shown in FIG. 15A. Just as the harmonics lie at theintersection of the underlying sinusoid and the alias sinusoid in thesinusoidal map, the harmonics lie at the intersection of these two,different spirals in the spiral map. In the upper row 1510 of FIG. 15A,harmonics (shown with triangles H_(A1), H_(A2), and the like) appear atthe intersection of the underlying sinusoid S_(U) and an alias sinusoidS_(A) with periods a and â as shown. The corresponding intersections oftwo spirals with constants of proportionality a and â can be seen inspiral plot 1516.

The ambiguity between the two spirals is resolved upon interpolatingalong the spiral 1518 of constant â, which selects the sinusoid andspiral shown in the lower row 1520 of FIG. 15A. FIG. 15B shows anexample in which F_(test) is just above the Nyquist rate for thesinusoid. Interpolation along each arm of the five-armed spiral 1548corresponds to interpolation along five shifted sinusoids S₁, S₂, S₃,S₄, S₅, of period â. In both cases, selecting â is useful, because,unlike a, â depends sensitively on F_(o)

Sinusoid Sampled Above but Near Nyquist Rate

Continuing special case 2, alias frequencies are not expected for thecondition F_(test)=mF_(o)(1−α), as long as m and α are chosen so thatF_(test)≧2F_(o). The sampling is sufficient to unambiguously determinethe period a-of the underlying sinusoid. However, the patterns 1410 a,1410 b, 1420 a, 1420 b, 1420 c shown in FIG. 14 that attract theattention of the eye consist of groupings of every m^(th) harmonic. Ifall but every m^(th) harmonic were completely removed (for example, n=m,2m, 3m, . . . or n=m+1, 2m+1, 3m+1, . . . ), the sampling period woulddrop by a factor of m, bringing the sampling rate below the Nyquistrate. The method for calculating the alias period for the separatedsub-set of harmonics is the same as in the previous section. One subsetof harmonics transforms to a single spiral arm that is part of anm-armed spiral whose constant of proportionality is equal to the newalias period as shown in FIG. 15B. As long as each arm is treatedindependently, the m groups of harmonics (the spiral arms) correspond tom shifted sinusoids. This provides a model for the patterns 1410 a, 1410b, 1420 a, 1420 b, 1420 c shown in the left column 1401 of FIG. 14.

Cutting Across the Underlying Curves

Audio signals often contain more than one set of harmonics. For example,two harmonic series with two different fundamental frequencies may bepresent in the spectrum when the test fundamental, F_(test), is chosento be well above both of these fundamentals. The second harmonic seriesonly increases the density of sampling and further nails down the valueof the period, a. However, when attempting to differentiate between thefundamental frequencies, the value of the period a does not provide thenecessary information to differentiate the frequencies. Visually, it isevident that it is not the underlying sinusoid but the effects relatedto visual dissonance—that is, the patterns shown in FIG. 14—which appearto change rapidly with either F_(o) or F_(test).

Additional points may be interpolated along the spiral paired with analias sinusoid, effectively cutting across the underlying curve. Thechoice of a transformation that increases the proximity of points alongthe alias sinusoid facilitates performing this interpolation. Cuttingacross the underlying curve forces a choice between the constants ofproportionality, a and â. Then, the extremely sensitive dependence of âon F_(test) makes it a useful diagnostic of fundamental frequency. Forexample, â=a/α becomes infinite as F_(test) passes through F_(o).

Note that when the eye follows the points of close proximity shown ineither column 1401, 1402 of FIG. 14, the eye is traveling along thecurves just specified which cut across the original, underlying curves.The eye is tracking changes in â as the curves move (in either map). Theeye can easily separate more than one harmonic series using the mapseven without additional interpolation because the eye groups togetherpoints of close proximity. Contour plotting algorithms introduceinterpolation numerically.

Application—Harmonic and Artifact Tracking

Harmonic tracking in the spectral analysis of an audio signal may bedemonstrated using the spectrum of an organ pitch produced with a 16foot flute stop. FIG. 16 shows the spiral map 1610 of the spectrum withthe test fundamental, F_(test), set near 65 Hz. The nominal fundamentalfrequency of the note C is 65.4 Hz. According to special case 1, theharmonics are expected to lie near the positive real axis. While theexpected harmonic series passes through the real axis, a second patternappears in a different part of the plane. The plot of the spiral map1610 shows a grouping of harmonics passing through the real axis, butalso shows a second set grouped into another spiral. An F_(test) scanrevealed that this set of harmonics moved to the positive real axis whenF_(test) was just above 69 Hz, which is the fundamental frequency of thenote, C sharp. The second pitch could have been in this data set due tothe reverberation characteristics of the recording hall.

Similar analysis has identified the presence of and spatially separatedharmonics of the line frequency, 60 Hz, from the main spectral sequenceand has been used to scan for harmonics present in office backgroundnoise.

Note also that the sinusoidal map applied to spectra containing aninharmonic sequence, such as that described by Equation 3 (rather thanpure harmonics) will have a chirp in â. Different inharmonicitiescorrespond to different chirp rates. The sinusoidal map of a frequencyspectrum can be used in this way to produce a sonification of the timbreof a musical instrument. Rather than converting image data to a serialdata set and using the intensity to create a function of time from theintensity, the method of the present invention takes the position of thepeaks and maps them onto the positions of the sinusoid. More generally,one may apply the sinusoidal map to any serial data set containing asequence of peaks, and sonify the data by creating a new signal in whichthe positions of the sinusoid samples are defined by the positions ofthe peaks in the data set, rather than by the peak intensities. Thesesignals will respond to the data near and below the Nyquist rate for theunderlying sinusoid. In essence, one would be listening to the alias andnot the underlying sinusoid. For example, a harmonic series in aspectrum would produce a pure tone. Below the condition F_(test)=F_(o),the pure tone would vary in frequency during a scan of F_(test). Wellabove the condition F_(test)=F_(o), the pure tone would be that of theunderlying sinusoid and would not change because the underlying sinusoidwould be sampled at a rate that satisfies the Nyquist criterion.

Two-Dimensional Data Analysis—Microscopy Images

The system and method of the present invention may be extended bysubstituting two-dimensional blocks (or arrays of data) for the parallellines. Each two dimensional block of data is repeated vertically ntimes. A target sequence of shift values is defined that is a functionof n and other parameters (the model). The n^(th) block is shifteddownward so that it intercepts the x-axis at the n^(th) value of atarget sequence of shift values. If there is a pattern in the data thatagrees or conforms to the model, the pattern will lie along the x-axis.When the methods of the present invention are applied to high resolutionmicroscopy images, the methods provide a way to simultaneously view highresolution information (along the y-axis) and macroscopic trends (alongthe x-axis).

An example is shown in FIG. 17A. High resolution images of an opticalgrating were acquired with a scanning electron microscope. A typicalimage 1791 is shown in FIG. 17A. A large number of additional highresolution images (not shown) were acquired, each starting at the rightedge of the previous image. A panoramic image was formed by stitchingthese images together, forming a new image. In the example, thirtycopies of the panoramic image were made and rotated such that the thirtycopies were all parallel to the y axis, intercepting the y axis at x=n,where n=1 to 30. Each copy was shifted downward by an amount y_shift(n),forming the image 1795 shown in FIG. 17B. In this example, the shift wasy_shift(n)=n x (test groove period). Motion was introduced by varyingthe test groove period. FIG. 17B shows the deviation between the grooveperiodicity and the model, and the presence of an artifact in the sample(error in groove spacing). By comparing images taken with differenttypes of microscopes (such as atomic force microscopes (AFM), scanningelectron microscopes (SEM), scanning tunneling microscopes (STM), andthe like) this technique may be used to separate certain types ofartifacts that are unique to the particular microscope used in theimages and those that are characteristic of the sample.

When the data is in the form of a three dimensional array (a volume witha value at each point in the volume), each volume is replicated multipletimes so that each copy is centered about one of a set of parallel linesthat that crosses the x-y plane as a grid. A target set of shift valuesis defined for each position on the grid. Each volume in the grid isthen shifted so that it that intercepts the x-y plane using the targetshift values. Scanning the definition of the target shift valuesproduces motion. The resulting animated volume may be viewed in 3-D, ora 2-D animated slice of the volume may be viewed.

3-D animations of serial data sets, such as frequency spectra, may alsobe created in a similar manner. Each frequency axis is replicatedmultiple times so that each copy lies along one of a set of parallellines in 3-D that cross the x-y plane as a grid. A target set of shiftvalues is defined for each position on the grid. Each line in the gridis then shifted so that it intercepts the x-y plane using the targetshift values. Scanning the definition of the target shift valuesproduces motion. The resulting animated volume may be viewed in 3-D, ora 2-D animated slice of the volume may be viewed.

While various embodiments in accordance with the present invention havebeen shown and described, it is understood that the invention is notlimited thereto. The present invention may be changed, modified andfurther applied by those skilled in the art. Therefore, this inventionis not limited to the detail shown and described previously, but alsoincludes all such changes and modifications.

1. A method of transforming a frequency spectrum data set from afrequency space to an animated rectangular coordinate space, the methodcomprising: mapping a frequency axis onto a curve, including:replicating the frequency spectrum into a plurality of lines; aligningthe plurality of lines in parallel in the rectangular coordinate spacesuch that the last line intercepts the horizontal axis of therectangular coordinate space at a horizontal coordinate equal to thenumber of the plurality of lines; shifting each line by an amount thatis a function of the number of lines in the plurality of lines to map atarget spectral sequence along the horizontal axis of the rectangularcoordinate space; mapping a value of a function of the frequencyspectrum data set onto a third dimensional axis of the rectangularcoordinate space; scanning a parameter of the frequency spectrum toproduce an animated contour plot sequence in the rectangular coordinatespace.
 2. The method of claim 1, wherein the curve is displayed in anx-y plane with a horizontal x-axis and a vertical y-axis.
 3. The methodof claim 2, wherein the vertical y-axis is limited to avoid redundancyin the contour plot sequence.
 4. The method of claim 2, wherein a targetspectral peak sequence is mapped along the x-axis.
 5. The method ofclaim 4, wherein additional target spectral peak sequences arepositioned consecutively along the x-axis.
 6. The method of claim 1,wherein the plurality of lines are of equal length.
 7. The method ofclaim 1, wherein the function of the frequency spectrum is at least oneof Fourier transform amplitude, phase, power, or spectral density andeach line is further shifted by an amount that is a function of thefrequency spectrum.
 8. The method of claim 1, wherein the plurality oflines are aligned vertically.
 9. A system for transforming a frequencyspectrum data set from a frequency space to an animated rectangularcoordinate space, the system comprising: a computer system, the computersystem including a plurality of modules including: a frequency axismapping module that maps a frequency axis onto a curve, including: acurve definition module that replicates the frequency spectrum into aplurality of lines, the plurality of lines including at least a firstline and a last line; a shifting module that aligns the plurality oflines in parallel in the rectangular coordinate space such that the lastline intercepts the horizontal axis of the rectangular coordinate spaceat a horizontal coordinate equal to the number of the plurality of linesand wherein each line is shifted by an amount that is a function of thenumber of lines; a frequency function mapping module that maps a valueof a function of the frequency spectrum data set onto a thirddimensional axis of the rectangular coordinate space; a scanning modulethat scans a parameter of the frequency spectrum to produce an animatedcontour plot sequence in the rectangular coordinate space.
 10. Thesystem of claim 9, wherein the frequency axis mapping module displaysthe curve in an x-y plane with a horizontal x-axis and a verticaly-axis.
 11. The system of claim 10, wherein the frequency axis mappingmodule further maps a target spectral peak sequence along the x-axis.12. The system of claim 9, wherein the frequency axis mapping modulefurther positions additional target spectral peak sequencesconsecutively along the x-axis.
 13. The system of claim 9, wherein theplurality of lines replicated by the curve definition module are ofequal length.
 14. The system of claim 9, wherein the frequency functionmapping module maps the function of the frequency spectrum that is atleast one of Fourier transform amplitude, phase, power, or spectraldensity.
 15. A computer program product comprising a non-transitorycomputer readable medium, the non-transitory compute readable mediumincluding computer-executable instructions for transforming a frequencyspectrum data set from a frequency space to an animated rectangularcoordinate space, the computer-executable instructions comprising:instructions for mapping a frequency axis onto a curve, including:instructions for replicating the frequency spectrum into a plurality oflines; instructions for aligning the plurality of lines in parallel inthe rectangular coordinate space such that the last line intercepts thehorizontal axis of the rectangular coordinate space at a horizontalcoordinate equal to the number of the plurality of lines; instructionsfor shifting each line by an amount that is a function of the number oflines in the plurality of lines to map a target spectral sequence alongthe horizontal axis of the rectangular coordinate space; instructionsfor mapping a value of a function of the frequency spectrum onto a thirddimensional axis of the rectangular coordinate space; instructions forscanning a parameter of the frequency spectrum to produce an animatedcontour plot sequence in the rectangular coordinate space.
 16. Thecomputer program product of claim 15, wherein the instructions formapping a frequency axis onto a curve further include instructions fordisplaying the curve in an x-y plane with a horizontal x-axis and avertical y-axis.
 17. The computer program product of claim 16, whereinthe instructions for mapping a frequency axis onto a curve furtherinclude instructions for mapping a target spectral peak sequence alongthe x-axis.
 18. The computer program product of claim 17, wherein theinstructions for mapping a target spectral peak sequence along thex-axis further include instructions for positioning additional targetspectral peak sequences consecutively along the x-axis.
 19. The computerprogram product of claim 15, wherein the instructions for replicatingthe frequency spectrum into a plurality of lines further includeinstructions for replicating the frequency spectrum into a plurality oflines of equal length.
 20. The computer program product of claim 15,wherein the function of the frequency spectrum is at least one ofFourier transform amplitude, phase, power, or spectral density.